How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN

Abstract Current language models can generate high-quality text. Are they simply copying text have seen before, or learned generalizable linguistic abstractions? To tease apart these possibilities, we introduce RAVEN, a suite of analyses for assessing the novelty generated text, focusing on sequential structure (n-grams) and syntactic structure. We apply to four neural trained English (an LSTM, Transformer, Transformer-XL, GPT-2). For local structure—e.g., individual dependencies—text with standard sampling scheme is substantially less novel than our baseline human-generated from each model’s test set. larger-scale overall sentence structure—model-generated as even more baseline, but still sometimes copy substantially, in some cases duplicating passages over 1,000 words long training also perform extensive manual analysis, finding evidence that GPT-2 uses both compositional analogical generalization mechanisms showing GPT-2’s usually well-formed morphologically syntactically has reasonably frequent semantic issues (e.g., being self-contradictory).